Skip to main content
Cornell University
We gratefully acknowledge support from the Simons Foundation, member institutions, and all contributors. Donate
arxiv logo > cs > arXiv:2306.07899

Help | Advanced Search

Computer Science > Computation and Language

(cs)
[Submitted on 13 Jun 2023]

Title:Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks

Authors:Veniamin Veselovsky, Manoel Horta Ribeiro, Robert West
View a PDF of the paper titled Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks, by Veniamin Veselovsky and 2 other authors
View PDF
Abstract:Large language models (LLMs) are remarkable data annotators. They can be used to generate high-fidelity supervised training data, as well as survey and experimental data. With the widespread adoption of LLMs, human gold--standard annotations are key to understanding the capabilities of LLMs and the validity of their results. However, crowdsourcing, an important, inexpensive way to obtain human annotations, may itself be impacted by LLMs, as crowd workers have financial incentives to use LLMs to increase their productivity and income. To investigate this concern, we conducted a case study on the prevalence of LLM usage by crowd workers. We reran an abstract summarization task from the literature on Amazon Mechanical Turk and, through a combination of keystroke detection and synthetic text classification, estimate that 33-46% of crowd workers used LLMs when completing the task. Although generalization to other, less LLM-friendly tasks is unclear, our results call for platforms, researchers, and crowd workers to find new ways to ensure that human data remain human, perhaps using the methodology proposed here as a stepping stone. Code/data: this https URL
Comments: 9 pages, 4 figures
Subjects: Computation and Language (cs.CL); Computers and Society (cs.CY)
Cite as: arXiv:2306.07899 [cs.CL]
  (or arXiv:2306.07899v1 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2306.07899
arXiv-issued DOI via DataCite

Submission history

From: Veniamin Veselovsky [view email]
[v1] Tue, 13 Jun 2023 16:46:24 UTC (308 KB)
Full-text links:

Access Paper:

    View a PDF of the paper titled Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks, by Veniamin Veselovsky and 2 other authors
  • View PDF
  • TeX Source
license icon view license
Current browse context:
cs.CL
< prev   |   next >
new | recent | 2023-06
Change to browse by:
cs
cs.CY

References & Citations

  • NASA ADS
  • Google Scholar
  • Semantic Scholar

1 blog link

(what is this?)
export BibTeX citation Loading...

Bookmark

BibSonomy logo Reddit logo

Bibliographic and Citation Tools

Bibliographic Explorer (What is the Explorer?)
Connected Papers (What is Connected Papers?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)
  • About
  • Help
  • contact arXivClick here to contact arXiv Contact
  • subscribe to arXiv mailingsClick here to subscribe Subscribe
  • Copyright
  • Privacy Policy
  • Web Accessibility Assistance
  • arXiv Operational Status